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ABSTRACT 

Construct validity of a prototype baccalaureate 
generic problem solving skills test was explored through a structural 
acalysis. The assessment method used three real-life problem 
situations crossed by six generic skills, namely Decision-Making, 
Communication, Analysis, Synthesis, Valuing and Execution. There were 
six written responses for each skill for each problem situation. The 
structural model included 18 skill/task observations with 6 latent 
skill variables, 3 latent trait variables and a higher order general 
factor (g). A multitrait-multimethod design was employed using a 
confirmatory factor analysis (LISREL VI). The data set moderately 
conformed to the model with most of the variation in performance 
attributed to the general factor. Factor loadings were neither 
consistent across skills nor across tasks. Therefore, generic skills, 
as measured by this procedure, possess little independence from the 
general factor. The weaknesses in the assessment method and 
implications for test methodology and research in the assessment of 
generic problem solving skills are discussed. Descriptions of the 
three real life problem situations are appended. (Author/LMO) 
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Abstract 

Construct validity of a prototype baccalaureate generic problem solving 
skills test was explored through a structural analysis^ The assessment 
method used three real-life problem situations crossed by six generic 
skills, namely Decision-Making, Communication, Analysis, Synthesis, 
Valuing and Execution. There were six written responses for each skill 
for each problem situation. The structural model included 18 skill/task 
observations with 6 latent skill variables, 3 latent trait variables and 
a higher order general factor (g). A mul t itrait-mul tiraethod design was 
employed using a confirmatory factor analysis (LISREL VI). The data set 
moderately conformed to the model with most of the variation in 
performance attributed to the general factor* Factor loadings were 
neither consistent across skills nor across tasks* Therefore, generic 
skills, as measured by this procedure, possess little independence from 
the general factor* The weaknesses in the assessment method and 
implications for test methodology and research in the assessment of 
generic problem solving skills are discussed* 
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Generic Problem Solving Skills: 
Are They Misnomers as Constructs of Educational Outcomes? 
Introduction 

While there has been considerable concern regarding the 
development and assessment of higher order cognitive processes (herein 
called generic problem solving skills) at all educational levels (NAEP, 
1981; Bloom, 1984; Frederickson, 1984; Ennis, 1985; Sternberg & Baron, 
1985), there is still controversy over the nature of such constructs 
and the structures of such abilities (Carroll, 1985; Heartel , 1985). 
More specifically, there is debate over whether such abilities are 
domain general or domain specific (Baron, 1982; Keil, 1982), or whether 
there is a hierarchical configuration including a higher order general 
factor and lower order specific factors (Gustafson, 1984; Carrol, 
1985). Such issues converge at the confluence of cognitive science, 
psychometr ics and education in which the first concerns their structure 
and development, the second their measurement, and the third the 
practice of helping individuals acquire them* 

This paper focuses on three perplexing issues undergirding the 
assessment of generic problem skills: 1) What are the structural 
relationships among such skills?; 2) Are the skills generic across 
problem solving tasks or are they specific to individual tasks?; and 3) 
To what degree are generic problem solving skills unique from one 
another or from a general factor? The answers to these questions may 
shed light on not only the construct validity of tests purporting to 
measure such entities, but also whether educational programs may 
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legitimately claim them as goals or desired outcomes. 

Generic problem solving skills may also be referred to as 
higher order mental processes (Bloom, 1984), critical thinking 
skills (Ennis, 1985), domain general skills (Keil, 1982), or 
reflective thinking skills (Dewey, 1933; Baron, 1981, 1982). They 
are considered as skills as opposed to capacities (Baron, 1982) 
since they are subject to the laws of learning (cue fidelity, 
reinforcement, practice, transfer) and can be acquired through 
educational or training experiences. Such skills are considered 
generic or ubiquitous in the sense they repeatedly occur in the 
analysis of skills underlying the resolution of real-life problems 
(Woditsh, 1977). Moreover, certain skills have been identified in 
a variety of contexts through a variety of research methods 
(Peterson & Wat kins, 1?79; Peterson, 1982). The most frequently 
occuring skills include Decision-Making, Communication, Analysis, 
Synthesis, Valuing and Execution. These skills, however, are 
subsumed under a general intelligence factor which represents the 
power of reasoning or the "noegensis" of abstract entities 
(Spearman, 1923), total life learnings (Carroll, 1982), or fluid 
intelligence (Horn and Cattell, 1966; Gustafson, 1984). 

One method used to assess generic problem solving skills is to 
measure them across a set of real-life problem situations analogous to 
the assessment center method (Moses & Byham, 1977). In this technique, 
an individual is presented with a real-life problem h'tuation (see 
Appendix I) and set of directive cues, one for each skill, designed to 
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elicit a written response that can be evaluated in terms of high, 
medium or low performance (Peterson, 1984). The directive cues were 
derived from general education competency statements (Peterson & 
Watkins, 1979) and are presented in Table 1. Such a test, while 
initially intended to assess the achievement of generic problem solving 
skills in postsecondary education, also provides an opportunity to 
investigate theoretical issues pertaining to the nature of such 
abilities. To date, the construct validity of the assessment center 
approach (tasks crossed by skills) has already been brought to question 
using exploratory factor analytic procedures (Sackett & Dreher, 1982). 

Place Table 1 about here 

A confirmatory factor analytic approach (Long, 1983; Bagozzi, 
1981, Gustafson, 1984) is employed in the present study to examine an 
a-priori hypothetical structure involving generic problem solving 
skills, a general intelligence factor and task factors as portrayed in 
Figure 1. According to the generic skills theory, skills should be 
present across most real-life problem solving situations (herein called 
tasks). If skills are truly generic, a statistical hypothesis 
would be that there would be greater common variance across skills 
than across tasks. Further, a general factor should capture common 
variance across both skills and tasks. Therefore, there should be 
consistently higher factor loadings across the skill factors than 
across task factors if the problem solving skills are generic 



6 



Generic Problem Solving Skills 

6 



rather than task specific. 

Place Figure 1 about here 
Method 

Subjects. A field test of a prototype generic problem solving 
skills test (Peterson, 1984) included 20 lower division students from 
introductory Psychology classes, 26 upper division students from Human 
Relations and ROTC classes and 16 graduate students from the College of 
Education at a large southeastern university. The lower division 
students received course credit for research participation, upper 
division students received a small stipend ($10) and graduate students 
met individual course requirements. 

Procedure. When subjects arrived for the prearranged testing 
sessions, they were informed they had unlimited time to complete their 
written responses to the three assessment tasks. Thus the speed factor 
as an element in the measurement of intellectual functioning was not 
present (Carroll, 1982). Most students completed their responses to 
the three problem solving tasks (i.e. situations) within two hours and 
no student took longer than three. Each task included six questions, 
one for each generic problem solving skill. The length of the written 
responses to each question varied from 10 - 250 words. 

The Situations. Three real-life problem situations (see Appendix) 
were developed to represent general academic content domains. For 
example, Cuban Crisis drew predominantly on perspectives from the 
social sciences and humanities. Land Development from the social 
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Table 1, Generic Problem Solving Skills and Directive Cues 
Used In Real-Life Situations 



Skills 



Directive Cues 



1 . Deci s i on-mak i ng 



Describe which course(s) of action 
you would recommend in the above 
situation. 



2. Communication 



As a (role specific such as a 

social worker), what is the central 
issue for you in the above situation? 



3. Analysis 



Describe the problem from A's 
perspective (i.e. a player), B's 
perspective and C's perspective. 



4. Synthesis 



List as many courses of action that 
might be taken including ones you 
woulld not choose to follow. 



5. Valuing 



What values underlie A's behavior, 
B's behavior, C's behavior? 



6. Execution 



For the course(s) of action you 
recommenced (in the decision making 
stimulus), outline the sequence of 
actions you would take to implement 
the solution. 
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Latent G Variable 



Latent Skill Variables 
(n = 6) 



Observed Skill 
X Task Variables 
(n = 18) 



Latent Task 
Variables (n = 3) 




Figure 1. Confirmatory factor model for the structure of generic problem solving skills. 
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sciences and natural sciences and Token Economy from the social 
sciences (especially Psychology) and professional schools* In each 
situation, the subject assumes a responsible social role such as a 
social case worker, a county commissioner, or a schoolboard member* 
The respective scenarios were intended to engage student interest and 
to be sufficiently challenging and complex so that the highest level of 
all six skills could be demonstrated* While not intending, the 
situations varied in degree of difficulty for four of six skills 
excluding Decision Making and Valuing (Peterson, 1985)* The full range 
of performances (1-5) were demonstrated for each skill for every task* 

Rating Procedures* For each of the three problem solving 
situations (tasks), there were six written responses, one for each 
skill directive* Each response was rated holistically on a 5-point 
continuum with anchors describing low (1), medium (3) and high (5) 
performances based on the works of Piaget (see Flavel, 1963), Kohlberg 
(1971), and Perry (1970). Each written response (18 per subject) was 
rated by 3 trained ABD doctorial students in Counseling Psychology* 
The interrater product moment correlations for 18 skill/task 
performances ranged from .35 to .73 with a mean of .53. The alpha 
reliability estimated for 3 pooled ratings for each of the 18 
performances ranged from .63 to .89 with a mean of .76 (Peterson, 
1984)* 

The test. As a comprehensive measure of generic problem solving 
skills, the test was able to differentiate levels of educational 
attainment* There was a step wise progression from lower division to 
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upper division to graduate level* The alpha reliability estimates for 
the six skill dimensions (9 ratings) were from *73 & *85 with a mean 
of *80. The range of intercorreltaions between skills was from *33 
to *66 with a mean of *55* Thus while there was a high degree of 
covariance among the skills, they would appear to be independent 
constructs possessing a substantial degree of discriminant validity* 
Generalizatibility coefficients '^for 3 tasks and 3 raters were from .51 
to .74 with a mean of .64 (Peterson, 1985). At this point, if there is 
a potential weakness in the test, it may be that the universe of 
problem solving situations was not adequately sampled (low 
generalizatibility coefficients) to affirm that generic skills scores 
derived from the test adequately sample a universe of scores. 

Method of Analysis. A multitrait-mul t imethod matrix was created 
with three tasks crossed by six skills. There were 18 observed 
skill/task variables* The structural model as proposed in Figure 1 was 
tested using a confirmatory factor analysis approach (Long, 1983). In 
this analysis, a general factor was designated as a latent variable as 
well as the six generic skill variables and three task variables. A 
confirmatory factor analysis permits the imposition of constraints on 
the data set by specifying which pairs of common factors are to be 
correlated and which observed variables and common factors are to be 
associated. The LISREL VI program (Joreskog & Sorbom, 1983) was used 
to conduct the analysis* 

It must be acknowledged, however, that the first order variables 
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of skills and tasks contain specific variance after the variance 
accounted for by the second order J factor has been partialled out* 
Different, but potentially viable, models could emerge when the latent 
J factor is omitted* Therefore a post hoc confirmatory analysis was 
conducted to determine whether the data also conforms to a model 
comprised only of tasks crossed by skills* 

Results 

One can note from the intercorrelation matrix presented in Table 2 
that the 18 skill/task performances are intercorrelated to a fairly 
high degree* Intercorrelations span from *58 to -*06 with a mean 
correlation of *32* Factor loadings (i*e* LISREL estimates of maximum 
likelihood) for the confirmatory factor analysis of proposed model are 
presented in Table 3* The goodness of fit for the whole model was 

= 114*36, df = 103, £ = .210, goodness of fit index of *841. These 
results suggest that the data set moderately conforms to the proposed 
model* The strongest and most consistent loadings occured on the 
general factor* Loadings of skills across tasks generally (with the 
exception of the Communication skill) entailed one high loading 
accompanied by two low loadings suggesting that these skills are not 
generic once the variance associated with the G factor is partialled 
out* Task factor loadings were also inconsistent indicating that tasks 
alone do not ::onsistently account for levels of performance* The 
interpretation of this analysis is that skills interact with tasks such 
that the effect of skills is not the same across tasks nor is the 
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effect of tasks the same across skills. 



Place 


Table 


2 


about 


here 




Place 


Table 


3 


about 


here 



Since the J factor appears to account for the major source of 
variation, a post hoc confirmatory analysis was conducted to determine 
whether a tasks crossed by skills alone model without the higher order 
J factor would be viable. The goodness of fit this model was 

= 239.66, df = 121, p = .000, goodness of fit index of .712. 
Therefore, the tasks by skills model was not confirmed by the data set. 

Discussion 

The present study sought to determine the structural relationships 
among generic problem solving skills, tasks and a general factor in 
response to real-life problem solving situations. The results of a 
confirmatory factor analysis indicated that dimensions of generic 
problem solving skills or higher order mental processes may not possess 
a high degree of independence from each other or from a general factor 
once variation due to a general factor has been partialled out. 
Differences in levels of performance across 18 observed performances (6 
skills X 3 tasks) can not be predicted from tasks crossed by skills 
alone. Therefore the conclusion is that there is little evidence to 
support that there are psychomet ical ly meaningful constructs as generic 
problem solving skills independent of the general factor in real-life 
problem solving situations. Only the constructs communication (what is 
the central issue?) and Execution (list steps to implement a solution) 
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Table 2 Intcrcorrelatlon matrix of 18 skill/task variables (n « 62) 



Dec - Dec - Dec - Com 1 Com 2 Com 3 Anal 1 Anal 2 Anal 3 Syn 1 Syn 2 Syn 3 Val 1 Val 2 Val 3 
Make 1 Moke 2 Make 3 



Decision making 1 KOO 



Decision making 2 


.31 


1-00 




























Decision making 3 


.44 


.32 


1.00 


























Communication 1 


.58 


.46 


.34 


1.00 
























Communication 2 


.29 


.43 


.25 


.49 


leCO 






















Communication 3 


.23 


.30 


.46 


.49 


.33 


1.00 




















Analysis 1 


.51 


.36 


.41 


• 48 


.37 


.40 


1.00 


















Analysis 2 


.23 


.24 


.30 


.22 


.18 


.37 


.37 


1.00 
















Analysis 3 


.44 


.21 


.40 


.27 


.12 


.35 


.33 


.23 


1.00 














Synthesis 1 


.38 


.34 


.37 


.37 


.34 


.27 


.50 


.20 


.32 


1.00 












Synthesis 2 


.36 


.39 


.24 


.25 


-26 


.28 


.18 


.43 


.26 


.19 


1.00 










Synthesis 3 


.36 


.31 


.46 


.34 


.39 


.18 


.48 


.27 


.35 


.36 


.43 


1.00 








Valuing 1 


.36 


.40 


.34 


.14 


.19 


.27 


.54 


.36 


.30 


.37 


.19 


.29 


1.00 






Valuing- 2 


.37 


.23 


.34 


.28 


-33 


.14 


.42 


.35 


.28 


.41 


.29 


.51 


.49 


1.00 




Valuing 3 


.46 


.31 


.39 


.43 


.43 


.27 


.58 


.38 


.26 


.41 


.35 


.53 


.36 


.48 


1.00 


Execution 1 


.45 


.27 


.36 


.47 


.15 


.41 


.45 


.39 


.53 


.33 


.27 


.40 


.19 


.38 


.32 


Execution 2 


.24 


.34 


.34 


.27 


.17 


.15 


.24 


.29 


.29 


.06 


.39 


.36 


.10 


.23 


.28 


Execution 3 


.10 


.13 


.30 


.03 


-.06 


.31 


.34 


.39 


.25 


.20 


.13 


.26 


.39 


.30 


.34 







Exec 1 


Exec 2 Exec 3 


Execution 


1 


1.00 




Execution 


2 


.37 


1.00 


Execution 


3 


.37 


.20 1.00 
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Table 3 - LISRtl Maximum Likelihood Estimates for Task by Skill Factor Matrix, 6 Factor Included 



Observed Skills i Decision Commcnlcatlon Analysis Synthesis Valuing Execution Task Task Task General 

Tasks Task Making Factor 

Factors 1 2 3 



Decision making 1 


• 117 












.451 






.599 


Decision making 2 


.083 














.265 




.478 


Decision making 3 


.776 
















.229 


.596 


Communication 1 




.580 










.635 






.519 


Coromunt cation 2 




.431 












.054 




.463 


CocRRiun I cation 3 




•480 














.785 


.466 


Analysis 1 






• 188 








.105 






.783 


Analysis 2 






.125 










.300 




.496 


Analysis 3 






-.588 












.223 


.529 


Synthesis I 








-.064 






.07! 






.597 


Synthesis 2 








.255 








.821 




.389 


Synthesis 3 








.724 










-.204 


.668 


Valuing 1 










.766 




-.185 






.615 


Valuing 2 










.134 






.035 




.621 


Valuing 3 










-.105 








-.130 


.722 


Execution \ 












.514 


.305 






.566 


Execution 2 












.258 




.258 




.371 


Execution 3 












.257 






.193 


.440 
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were there factor loadings greater than .250 across all three tasks. 
Perhaps Carroll (1985) is correct when he stated that factors naturally 
differ in the range of their applicability to cognitive tasks. 
Extending this thought, generic skills may function as potential rather 
than manifest entities as requirements of specific tasks beyond J 
(Baron, 1982). 

However, before adopting a reinterpretation of the notion of 
generic problem solving skills is noted by many prominent theoreticians 
(Dewey, Baron, Bloom), there are several issues in the measurement of 
such constructs that should be considered. The first is that there are 
inherent biases toward inflating the contribution of J in this method 
of assessment. The productions are all written responses, and thus, the 
level of skill performance across all skills could be highly influenced 
by attributes of general writing ability (eg vocabulary, spelling, 
syntax, and fluency). Therefore, further research is needed to control 
for the contribution of general writing ability, possibly by 
diversifying its method of performances perhaps along the line of 
Guilford's (1967) Product dimension in his model of general 
intell igence. 

A second consideration is that the number of tasks used, three, 
was too small to allow subjects to demonstrate the "genericness" of 
their problem solving skills. The results of a general izitbil ity study 
(Peterson, 1985) on the present data set suggest that as many as 10 
tasks may be required to achieve general izatibil ity coefficients 
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consistently above .80 across all skills. Third, the study requires 
replication with considerably more subjects than the present study 
(n = 62). In confirmatory factor analysis, the risk of Type I error is 
increased with a small sample size. 

Given the fact that the test appears to differentiate levels of 
educational attainment when IQ was equated (Peterson, 1984), the 
general factor here may also be indicative of general cultural learning 
as noted by Carroll (1982) and Undheim (1981) rather than as a 
pyschologically determined characteristic as fluid intelligence 
(Gustafson, 1984). General knowledge would appear to be very important 
in responding to real-life situations. 

The question posed in the title of this paper was whether generic 
problem solving skills are misnomers as constructs of educational 
outcomes. The answer to this question, based on the present 
findings, is that they indeed are misnomers given the method of 
assessment used in this study. It would appear that little is 
gained over the information a vocabulary test, a general knowledge 
test, or general reasoning test would provide. Generic problem 
solving skills may, however, be demonstrated with a greater variety 
of productions over more problem situations. Until then, one could 
not claim that generic problem solving skills are being measured 
much less attained by employing several short essays over several 
problem situations. The implication for test methodology is that a 
comprehensive and diverse test battery would be required to 
demonstrate whether such educational outcomes are being achieved. 

mrteoPiMMMiM 
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First, however, it must be demonstrated that such entities exist 
beyond the general factor alone. 
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APPENDIX 
Problem Situations 

Cuban Crisi s ( I ) 

As a case worker for the Department of Social Services, you have 
been assigned to work with a young Cuban woman, found beaten and semi- 
conscious in a Cuban neighborhoods Taken by police to a hospital, she 
acknowledged that she had been sexually assaulted, but refused to 
identify her attacker; she became hysterical at the suggestion that 
she undergo a physical examination to determine if rape had occurred. 
Her family has refused to cooperate with police attempts to investigate 
the presumed rape, although the police have been abfe to determine her 
attacker's identity by informants in the Cuban community. The girl*s 
father has flatly refused to swear out a warrant, and the girl refused 
to acknowledge whether that a rape has occurred or her assailant's 
identity. Frustrated, the police have called you to come to the police 
station and take over the case. 



Land Development (II) 

You are a County Commissioner and a special meeting has been 
called in order to settle a dispute that has occurred concerning the 
proposed development of 100 acres of land along a primitive river bank. 
The land in question is 25 feet beyond a marshy area that borders the 
river. The developer plans to subdivide the land into one acre 
residential tracts. The site is in a rural county that lies just 7 
miles south of a rapidly growing metropolitan area. The river and 
surrounding areas are noted for their excellent fishing and camping, 
annually attracting sportsmen and campers statewide and beyond. A 
major controversy has been raging ever since the developers intentions 
were made public. Local environmental groups here opposed the 
development while local businessmen have stressed the need for such a 
development. The developer had promised to delay the actual 
construction until environmental groups completed an environmental 
impact study. The special meeting has been called because several days 
ago the developer began construction of a road along the river bank and 
has also begun digging wells and septic tanks. The local environmental 
groups are demanding an immediate halt to construction. The developer 
claims that the construction delay is costing him thousands daily and 
that he intends to proceed with construction until concrete evidence is 
provided that the subdivision would be other than beneficial to the 
community. 
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Token Economy ( III ) 

You are a School Board member in an inner city school district* 
Disciplinary problems in the 3rd through 6th grades have tripled in the 
past year. The School Board has received numerous demands from parents 
to do something to restore order to the schoolrooms. Because of 
understaf f ing the classrooms are overcrowded, and it has become 
increasingly more difficult for teachers to manage the students. 
Fighting, practical joking, inattentiveness, the use of vulgar language 
and cheating abound. Several principals have asked the School Board to 
institute a token economy in all elementary school classrooms within 
the school district. Since this proposed solution v^as introduced at 
the last board meeting a controversy has raged between opposing 
factions in the P.T.O. Those favoring the institution of a token 
ecomony claim it is the most efficient way to eliminate undesirable 
classroom behavior. Those opposing the use of a token economy do so 
primarily on ethical and moral grounds. The board meets once next 
month to make recommendations for further courses of action. 
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